Integration of Face and Voice Recognition

cepstral features and features based on a bio–mechanical model of the visible articulators will be the identity–carrying characteristics extracted from acoustic speech and visual speech respectively. Speakers will be modelled by multi–layer perceptrons trained as discriminative models or, alternatively, as predictive models. In the discriminative modelling scheme, each speaker model will be trained to recognize its allocated speaker directly from his acoustic and visual speech. A measure, based on the cross–correlation between the motion of visible articulators and the acoustic speech, will be used for detecting the impostors who would use facial images and voice not originating from the same person (e.g. by using tape–recorded voice and mimicking the movement of visible articulators). The predictive modelling scheme is based on the belief that acoustic and visual speech are cross–correlated. Hence, one may be predicted from the other. Another assumption is that the mapping from acoustic speech to visual speech is speaker–specific; hence, each speaker will be modelled by an MLP trained to perform acoustic–to–visual speech mappings (prediction) for his speech. The prediction error for each speaker model will then act as the recognition measure. The lower the error, the better the model fits the given acoustic and visual speech. CONCLUSION Preliminary investigations have shown that person recognition accuracy can be improved by the joint use of vocal and facial information. It remains to be seen whether acoustic speech used in conjunction with visual speech will also yield improved recognition accuracy.